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Introduction and summary 


The White House will soon unveil a final version of its “college scorecard” — an 
online tool giving college-bound students and their families a hype-free snapshot 
of reliable information about any U.S. campus: real costs, graduation rates, student 
debt statistics, and earning potential of graduates. 

The college scorecard is a good idea and it has the potential to make college- 
bound students smarter consumers. The scorecard is part of a major effort by the 
White House and the U.S. Department of Education to understand and improve 
the college selection process. At a time when student loan debt has exceeded $1 
trillion, 1 fewer than 60 percent of college freshmen graduate within six years, 2 and 
the wages of recent grads have declined by nearly 5 percent since 2007, 3 it’s more 
important than ever that students make good decisions about where to go, what 
to study, and how to pay. But to help students make better decisions, the scorecard 
must be easy to understand and relevant to their decision-making processes. 

Though policymakers are working diligently and conscientiously to design a 
scorecard that will help students and families, the college scorecard has not been 
subjected to systematic testing by actual students and parents. Unfortunately 
this is typical of many disclosures government agencies require in the hopes of 
improving consumer choice. (CAP has previously written about a similar problem 
with the Securities and Exchange Commission’s recent revamp of disclosures that 
money managers must provide to prospective clients. 4 ) Without consumer test- 
ing, disclosures risk being overlooked and misunderstood. 

At the White House’s invitation, many college admissions and financial aid 
experts, including some from CAP, are weighing in on the college scorecard 
design. These experts are making every effort to put themselves in the shoes of 
prospective college students and are scrutinizing the draff scorecard for poten- 
tially confusing language or missing information. 
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But designing an effective information sheet 
about college costs, debt, and graduation rates 
is hard without feedback from actual users. 
Consider one student’s reaction to the draff 
scorecard after it had been through several 
rounds of experts: “What am I looking at? It 
looks like a bill or something but I’m not sure 
what it is,” said Kendra, a high school student, 
after examining a sample college scorecard. 

“This is why I hate college stuff.” 

Kendra’s comments show that even the best- 
intentioned policymakers can miss the mark. 
And if they do, students like Kendra may not 
give the college scorecard a second glance. 

This report uses the government college score- 
card project as an opportunity to explore how 
testing might lead to more effective disclosures. 
We took the college scorecard to college-bound 
high school students, asking them for feedback on 
design, content, and overall effectiveness. In the 
pages that follow, we discuss the findings of these 
focus groups, make recommendations specific 
to the college scorecard project, and draw some 
overall recommendations for improving the read- 
ability and usability of government disclosures. 

Here’s a summary of our recommendations. 


FIGURE 1 

The government scorecard 

A sample look at the Obama administration's draft scorecard 


University of the United States (UUS) 

Any town. State x Primarily Awards Bachelor's Degrees x Carnegie and Control 


What will it cost to attend UUS? 

Depending on your circumstances: 

Tuition and fees only 

In-state: $X,XXX 

Out-of-state: $X,XXX 

Tota I costs befo re a id : $XX,XXX-XX,XXX 


♦ Institutions that enroll similar types of students 


Costs 

Average net price after grants and scholarships compared to other 
institutions 


Average total costs after grants 
and scholarships ("net price"): $XX,XXX 

Change from last year: "fr X% 


UUS 


$- $5,000 $10,000 $15,000 $20,000 $25,000 $30,000 


How likely am I to graduate and how 
long will it take? 

• XX% of students graduate in 4-years 

• XX% of students graduate in 5-years 

• XX% of students graduate in 6-years 

• XX% of students transfer to another 
institution 


Graduation 

Percentage of full-time students who graduate within 6 years 
compared to other institutions 

UUS 

■ ■■ ■ — r 

0 20 40 60 80 100 


Will I be able to repay my student loans 
after I graduate? 

Former students are successfully 
repaying XX% of the total amount of 
federal student loans they took out to 
attend UUS. 


Student Loan Repayment 

Percentage of total loan amounts being repaid by former students 
compared to other institutions 

UUS 

♦ ♦♦ i — f 

0 20 40 60 80 100 


How much debt will I have when I 
graduate? 

We don't have this information yet. Before 
you enroll, ask UUS to tell you about how 
many students graduate with debt and how 
much they typically owe. 


Will I be able to get a job after I 
graduate? 

We don't have this information yet. Before 
you enroll, ask UUS to tell you about how 
many of their graduates get jobs, what 
kinds of jobs they get, and how much they 
typically earn. 


Student Loan Debt 

The average amount of loans students borrow to get a degree as 
compared to other institutions. The Administration is seeking ways 
to provide this information to students and families. 


Earnings Potential 

The President has proposed providing this information to students 
and families. 


General recommendations 

• Congress can encourage better disclosure through the way it legislates the devel- 
opment of effective disclosures by delegating to an agency both the authority to 
design the disclosure and the responsibility to prove its effectiveness through 
consumer testing. 
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• The White House should require agencies to test their disclosures, and it should 
ensure that research on disclosure design and efficacy is available to the public. 

• Whenever possible, agencies should use standard, commonly used terms in 
disclosures to promote better understanding of confusing terms and concepts. 


Selected college scorecard recommendations 

• The scorecard should include an introductory description, name, or logo that 
immediately communicates its purpose. 

• The scorecard should be redesigned by professional graphic designers to 
improve visual hierarchy for readability. 

• The government should test ways of communicating the confusing concept of 
“net price” and adjust the scorecard accordingly. 

• The scorecard should emphasize four-year graduation rates, not six-year rates if 
further testing confirms that the shorter timeframe is more relevant to students’ 
decision-making. 

• The government should develop alternative measures of student debt that mat- 
ter to students if further testing confirms that traditional measures such as repay- 
ment rate or default rate are not meaningful to students. 

• The online version of the college scorecard should include links to other out- 
comes such as graduates’ average salary and employment outcomes by major or 
department. 

• In general, the government should subject the college scorecard and other 
college-choice communication initiatives to rigorous testing — including focus 
groups, cognitive interviews, and surveys of parents and students — and make 
changes accordingly. 

The college scorecard is a key part of the Obama administration’s broader commit- 
ment to providing students and families with useful data in the college decision- 
making process. We hope the suggestions contained in this report are helpful 

contributions to this important project. 
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Testing the college scorecard 


The key to creating a usable disclosure is testing its efficacy with actual consum- 
ers. The Obama administration is developing a “college scorecard” — essentially a 
one-page disclaimer of key college cost and performance data. The administration 
describes the scorecard as “a new tool . . . that would assist prospective students 
and their families in comparing colleges before they choose using key measures 
of college affordability and value. The purpose of the tool is to make it easier for 
students and their families to identify and choose high-quality affordable colleges 
that provide good value.” 5 

Obviously the scorecard will only be useful if students understand it and find 
it relevant to their decisions. And the only way to know for sure if it fulfills its 
purpose is to ask students directly: Will you use this? Do you trust it? Is it easy to 
understand? What about it is helpful? What’s not? How would you change it? 

Government agencies can test disclosures in several ways. Qualitative testing 
through both focus groups and so-called cognitive interviews can help give 
designers an idea of how consumers react to the disclosure. 

Focus groups, in which a panel of sample users discuss and interact with the tested 
product, are a simple way to elicit a variety of reactions to the overall design and 
content of a disclosure. Cognitive interviews allow for a deeper understanding of 
the reader’s experience with the disclosure. In one-on-one cognitive interviewing, 
the participant thinks aloud while reading through the document and responds to 
questions that test the participant’s comprehension and preferences. Though focus 
groups and cognitive interviews typically involve small samples, a study by Robert 
Virzi of Raver Consulting showed that interviews involving just five participants 
could reveal 80 percent of usability issues, and 1 0 participants would yield 90 
percent of issues. 6 

Quantitative methods can also be used to test comprehension of disclosure ele- 
ments as well as their effect on consumers’ decisions. In 2007 the Federal Trade 
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Commission, for example, used controlled quantitative testing to evaluate the 
efficacy of mortgage disclosures. 7 Participants were asked a series of questions that 
addressed whether the disclosures conveyed key information. The results showed 
that consumers failed to comprehend key concepts that the mortgage disclosures 
were intended to convey. The commission has since advocated for improvements 
to existing mortgage disclosures. 8 

In many cases, government agencies use only focus groups and cognitive inter- 
views to test their draff disclosures, probably because the cost of large-scale 
quantitative studies would be prohibitive. Though a comprehensive study using 
multiple methods would be ideal, budgetary realities often constrain the scope of 
research. Our focus groups show that policymakers can get a taste of the problems 
inherent in disclosure design through small-scale research projects. Nevertheless, 
we recommend that Congress invest in better research to assure the success of 
their disclosure efforts. 

In the spring and fall of 2012, CAP took the college scorecard out for a test drive 
to see how students would react to its content and design. To give students some- 
thing to compare with the original government draff, we created our own revised 
version of the college scorecard by applying some common principles of disclo- 
sure design, (see Appendix on page 16) 

We showed the government scorecard and our own version to four different 
groups of college-bound high school juniors and seniors. To make it easier for 
students to evaluate the scorecard, where applicable, we filled it in with data about 
an actual university. 

It’s important to note at the outset that our focus groups were meant to be a start- 
ing point for research on the usability of the scorecard, not a definitive study. To 
completely uncover the changes necessary in the college scorecard, one would 
need to conduct more thorough research, including tests involving parents and 
nontraditional students, as well as one-on-one cognitive interviews. Nevertheless, 
these preliminary conversations do reveal some insights into how students react to 
the scorecard and some of the potential problems with its content and design. 
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FIGURE 2 

Comparing the government and CAP scorecards 

A side-by-side look at the government and CAP versions of the college scorecard 


University of the United States (UUS) 

Any town, State A Primarily Awards Bachelor's Degrees Carnegie and Control 


What will it cost to attend UUS? 

Depending on your circumstances: 

Tuition and fees only 

In-state: $X,XXX 

Out-of-state: $X,XXX 

Total costs before aid: $XX,XXX-XX,XXX 


Costs 

Average net price after grants and scholarships compared to other 
institutions 


Average total costs after grants 
and scholarships ("net price"): $XX,XXX 

Change from last year: X% 


$5,000 $10,000 $15,000 $20,000 $25,000 $30,000 


How likely am I to graduate and how 
long will it take? 

• XX% of students graduate in 4-years 

• XX% of students graduate in 5-years 

• XX% of students graduate in 6-years 

• XX% of students transfer to another 
institution 


Graduation 

Percentage of full-time students who graduate within 6 years 
compared to other institutions 

UUS 

I III — — « 

0 20 40 60 80 100 


Will I be able to repay my student loans 
after I graduate? 

Former students are successfully 
repaying XX% of the total amount of 
federal student loans they took out to 
attend UUS. 


Student Loan Repayment 

Percentage of total loan amounts being repaid by former students 
compared to other institutions 


UUS 



0 20 40 60 80 100 


How much debt will I have when I 
graduate? 

We don't have this information yet. Before 
you enroll, ask UUS to tell you about how 
many students graduate with debt and how 
much they typically owe. 


Student Loan Debt 

The average amount of loans students borrow to get a degree as 
compared to other institutions. The Administration is seeking ways 
to provide this information to students and families. 


Will I be able to get a job after I 
graduate? 

We don't have this information yet. Before 
you enroll, ask UUS to tell you about how 
many of their graduates get jobs, what 
kinds of jobs they get, and how much they 
typically earn. 


Earnings Potential 

The President has proposed providing this information to students 
and families. 


COLLEGE PRICE AND PERFORMANCE INFORMATION 
FROM THE U.S. DEPARTMENT OF EDUCATION 

University of the 
United States (UUS) 

123 Main Street, Anytown, State 12345 
(123) 456-7890 • financialaid@uus.edu 



QUICK LOOK 


Q 

e 


COSTS 

SXX.XXX: the average cost 
students pay out of pocket 

GRADUATION 

XX% of students graduate 

STUDENT LOAN DEFAULT 
XX% of students fail to pay 
their loans within two years 
of leaving school 


STUDENT LOAN DEBT 

SXX.XXX is the average amount 
of loans students borrow 


COSTS: What will it cost to attend UUS per year? 


Tuition and fees SX.XXX 

+ Living expenses, books $XX,XXX 

= Total costs for average student 

$XX,XXX 

- Average grants and scholarships 

SX.XXX 

= Average out of pocket cost 

$XX,XXX 


How UUS’ average out-of-pocket cost compares to other colleges 
Other university 


Other university 



GRADUATION: How likely am I to graduate and how long will it take? 


XX% of students graduate in 4 years 
XX% of students graduate in 5 years 
XX% of students graduate in 6 years 
XX% of students transfer to another 
institution 


Percentage of full-time students who graduate within 4 years 

UUS Other university 

I Other university I Other university 


S) STUDENT LOAN DEFAULT: Will I be able to repay my student loans after I graduate? 


XX% of former UUS students fail to 
repay their student loans within 2 
years of leaving UUS. 


Percentage of UUS students who default on their student loans 


Other university 



Other university 
| Other university 


UUS 



(j^P) STUDENT LOAN DEBT: How much will I owe when I graduate? 


The average student graduates with 
$XX,XXX in federal student loans 
- that's $XXX each month in loan 
payments for ten years. 

XX% of students at UUS take out 
student loans 


The average amount of loans students borrow to get a degree 

Other university Other university 


Other university 


Students' overall reaction to the scorecard 

After showing students copies of both the government and CAP versions of the 
college scorecard, we began with general questions about its purpose and useful- 
ness, such as “What’s the purpose of this document?” Students often met our 
questions with a blank stare. It became clear that Kendra’s initial reaction — “What 
am I looking at?” — was a common one. 

Other students initially felt that they understood the scorecard because they 
understood individual elements such as costs and graduation rates, but found they 
could not describe the scorecard’s purpose. One student asked, “Is this supposed 
to explain something about college?” 
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The government draft lacked the contextual information that would immediately 
clue the reader into what the disclosure is meant to tell them. Likewise; CAP’s 
revised version of the college scorecard lacked any introductory content that would 
initiate the reader into the disclosure. This is something a redesign should address. 

We also noticed that students read the government-designed scorecard in a 
nonlinear way. They were drawn to certain elements of the disclosure such as the 
tuition and fees numbers, graduation rate, and earnings potential. But when we 
asked students about other items on the government scorecard such as the percent 
change in cost from the previous year, students responded that they had missed 
that information. It maybe that the scorecard lacks essential design elements like 
a single focal point and a visual hierarchy that leads the reader from one part of the 
disclosure to another. CAP’s scorecard fared a little better on this count because of 
a clearer visual hierarchy. 

Another frequent complaint about both the government and CAP versions was 
that they did not include enough information. Focus group participants consis- 
tently said they wanted a one-stop shop for all of their college information needs 
and that the content of both scorecards was insufficient. 

The desire for more information was underscored by the comment made by one 
student who loved the college scorecard precisely because it filled his one-stop shop 
needs: “It’s everything I was looking for in a much easier to see format. [Without the 
scorecard,] I had to find it all — look here, look somewhere else, then look some- 
where else. This is great, especially if you’re trying to compare across schools.” 

Another general reaction to both the government and CAP scorecards was a 
desire for customization. Students talked about this in many different ways — for 
example, one student commenting on the estimated net price figure, said: “For 
me, it’s almost irrelevant. You have to add in your own family income. If you make 
$20,000 per year, this is way skewed. It must just take all incomes and all schools 
and lump it together, so I don’t know where I fall in that.” Some students asked for 
job placement numbers by major while others asked for graduation rates by race 
and ethnicity. Another student simply said, “I don’t really trust averages.” 

Next, let’s look at student feedback to different elements of the scorecard. 
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Reaction to cost information 


Students responded well to the inclusion of information pertaining to cost on the 
college scorecard. But when we pressed them about whether they understood the 
displayed cost elements; we 

FIGURE 3 

College costs on the government scorecard 

Students were substantially confused over cost terminology 


What will it cost to attend UUS? 

Depending on your circumstances: 

Tuition and fees only 

In-state: $X,XXX 

Out-of-state: $X,XXX 

Total costs before aid: $XX,XXX-XX,XXX 

Average total costs after grants 
and scholarships ("net price"): $XX,XXX 

Change from last year: "fr X% 


Costs 


found substantial confusion 
over terminology. One par- 
ticipant noted that the govern- 
ment-designed scorecard lists 
tuition and fees as components 
of cost but does not mention 
any other components, such as 
living expenses or books. This 
led the student to question just 
what was included in the “total 
costs before aid” figure that fol- 
lows the tuition and fees price. 

CAP’s version of the scorecard 
reflects an attempt to show all 
of the elements included in 
total costs in a simple mathe- 
matic formula: Tuition and fees 
+ living expenses and books = 

total costs. Students liked this 

presentation but they still ques- 
tioned our terminology. One student said, “I think room and board is different from 
living expenses. When I think of living expenses, I think of sheets and comforters.” 


♦ Institutions that enroll similar types of students 


Average net price after grants and scholarships compared to other 
institutions 


$ 5,000 $ 10,000 $ 15,000 $ 20,000 $ 25,000 $ 30,000 


FIGURE 4 

College costs on the CAP scorecard 

CAP'S cost formula: Tuition and fees + living expenses and books = total costs 


©: COSTS: What will it cost to attend UUS per year? 


Tuition and fees $X,XXX 

+ Living expenses, books $XX,XXX 

= Total costs for average student 

$XX,XXX 

- Average grants and scholarships 

$X,XXX 

= Average out of pocket cost 

$XX,XXX 


How UUS' average out-of-pocket cost compares to other colleges 
Other university 
UUS 

| Other university 


Other university 


$16,000 


$20,000 


$24,000 


Students also expressed confusion about the meaning of the terms used to express 
net price on both the CAP and government designs of the scorecard. When we 
asked students what the terms “net price” and “out-of-pocket cost” meant, students 
most often responded with blank looks or general comments such as “It includes 
everything, altogether.” And most students could not decide whether the net price 
or out-of-pocket figure would include the amount a student would take out in loans. 


It seems that students’ confusion about the terms used on the college scorecard 
is compounded by the inconsistency of terminology used throughout the college 
application process. Several students expressed a desire to have the same wording 
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used on all college enrollment and financial aid paperwork, from college applica- 
tions to the federal student financial aid forms to the scorecard itself. One student 
pointed out that colleges often use the term “total cost of attendance” and he was 
unclear which figure on the college scorecard corresponded to that term. “Can’t 
they just use the same words?” he asked. 

One surprising finding from the focus groups is that some financial aid jargon has 
so penetrated the college-search process that some students actually preferred 
jargon over plain language because it’s familiar. One student said she wished that 
the scorecard would use the term “EFC,” or expected family contribution, which 
appears on the Free Application for Federal Student Aid, or FAFSA. “I think 
everyone should use the same terms, because on your FAFSA they say EFC. I 
think they should use the same thing here.” 

This student’s comment reflects an even more complicated problem — a misun- 
derstanding of estimated family contribution, which is the government estimate of 
what a family will pay, but not necessarily what a college expects a family to pay. 


Reaction to graduation rate information 


The graduation rate was easily 
understood by all students 
and many found it to be a 
relevant measure of college 
quality. Both the government 
and CAP versions display a 
four-, five-, and six-year gradu- 
ation rate, but the government 
emphasizes the six-year rate 
while CAP highlights the 
four-year rate. 

Students were generally 
more interested in a four-year 
graduation rate than a six-year 
one. While students said they 
were aware that many college 
students take longer than four 


FIGURE 5 

Graduation rates on the government scorecard 

Government emphasizes the six-year graduation rate 

How likely am I to graduate and how 
long will it take? 

• XX% of students graduate in 4-years 

• XX% of students graduate in 5-years 

• XX% of students graduate in 6-years 

• XX% of students transfer to another 
institution 


FIGURE 6 

Graduation rates on the CAP scorecard 

CAP emphasizes the four-year graduation rate 

GRADUATION: How likely am I to graduate and how long will it take? 


XX% of students graduate in 4years Percentage of full-time students who graduate within 4 years 


XX% of students graduate in 5 years 
XX% of students graduate in 6 years 
XX% of students transfer to another 
institution 



UUS 

Other university 

Other university 

Other university 


0 

20% 


40% 

60% 80% 


Graduation 

Percentage of full-time students who graduate within 6 years 
compared to other institutions 

uus 

* »» ♦ twiimici iii wyi ii ^b wwiF ♦ 

0 20 40 60 80 100 
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years to graduate, every student we spoke to expected to graduate in four years. 
Indeed, some said that graduating in four years was an absolute must due to finan- 
cial concerns. As a result, students discounted the graduation rate scale used on the 
government scorecard because it did not seem applicable to their decision making. 


Reaction to student debt information 


The Obama administration’s version of the college scorecard includes two mea- 
sures that offer information about student debt. The first measure, “student loan 


repayment,” was incompre- 
hensible to our focus group. 

It refers to the percentage of 
a student’s total outstand- 
ing student loan debt that 
is in repayment. When we 
explained the concept, most 
students believed that this 
information would be irrel- 
evant to their decisions. 

CAP’s version of the scorecard 
replaced this measure with 
one showing the percentage 
of students who default on 
their loans. The reviews of this 
measure were mixed. Some 
students did not know the 
meaning of the term “default.” 
Others said that default rates 
are useful because they reflect 
whether graduates “get good 
jobs.” But several students 
commented that student loan 
defaults say more about the 
students than they do about 
the school. 


FIGURE 7 

How the government scorecard addresses the likelihood of loan repayment 


Students did not understand the "student loan repayment" measure 


Will I be able to repay my student loans 
after I graduate? 

Former students are successfully 
repaying XX% of the total amount of 
federal student loans they took out to 
attend UUS. 


Student Loan Repayment 

Percentage of total loan amounts being repaid by former students 
compared to other institutions 

UUS 

♦ ♦♦ | ♦ 

0 20 40 60 80 100 


FIGURE 8 

How the CAP scorecard addresses the likelihood of loan repayment 

Students had a mixed response to the "student loan default" measure 



XX% of former UUS students fail to 
repay their student loans within 2 
years of leaving UUS. 


Percentage of UUS students who default on their student loans 

Other university UUS 

Other university | Other university 

0 1% 2% 3% 4% 


5% 


FIGURE 9 

Another student debt measure on the CAP scorecard 

Students responded better to the "student loan debt" measure on both the CAP 
and government scorecards 


) STUDENT LOAN DEBT: How much will I owe when I graduate? 


The average student graduates with 
$XX,XXX in federal student loans 

The average amount of loans students borrow to get a degree 


- that's $XXX each month in loan 

Other university 

Other university 


payments for ten years. 

XX% of students at UUS take out 

UUS 


Other university 

student loans 

$15,000 $17,500 $20,000 

$22,500 $25,000 

$27,500 
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Students expressed much more satisfaction with the fourth measure on the CAP 
scorecard; which shows the average amount of debt a student will accrue while earn- 
ing a degree. Most students expected to take on student loan debt to pay for college, 
and they felt that the amount of debt would be relevant to their choices. “I think it’s 
kind of scary when you look at it,” said one student. “My father and I were talking 
about it and he said $20,000 is not that bad. It’s scary, but it could be worse.” 

Many students also like the monthly payment breakdown included on CAP’s 
version of the scorecard. “You can’t really comprehend how much $50,000 is until 
you have to pay for it every month,” said one student. “You don’t know how it’s 
going to fit into the job and life you’ll have when you graduate.” 

For another student, the monthly payment information was a wake-up call 
highlighting the reality that graduates end up paying their student debt for years. 
“It makes you think how long it would take to pay back. ... I don’t want to pay 
student loans for 10 years.” 


Reaction to employment information 

All of the focus group students we spoke with were enthusiastic about the inclu- 
sion of information about earnings potential on both the government and CAP 
college scorecard drafts. Students felt that the opportunity of securing a good job 
upon graduation was a major consideration and one about which they had little 
information. As one student put it, “I want to make mad dollars.” 

The main reaction we observed to the prospect of earnings information on the- 
college scorecard was “Give us more.” Students expressed an interest in average 
salary information broken down by major, as well as statistics on the percentage of 
students who obtain employment within one year of graduation. 

One student said this information would be helpful not only in evaluating the qual- 
ity of a college but also in putting the average loan debt information into context. 
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Steps needed to improve the scorecard 


Though more research can and should be done on the usability of the college 
scorecard; our focus groups suggest a few areas in need of improvement. We urge 
the Obama administration to incorporate the following changes in its final draft of 
the scorecard and continue testing them on students and parents: 

• Introduce readers to the scorecard with a simple and descriptive name, logo, 
or introductory sentences. Our focus groups suggested that the purpose of 
the college scorecard is not readily apparent to students. Without signals that 
are attention grabbing or explain the scorecard’s usefulness, there is a risk that 
students will either ignore it or fail to recognize its relevance to their college 
choices. The Obama administration should test ways to introduce students to 
the scorecard. These tests should include consideration of whether “college 
scorecard” is a sufficiently descriptive name for the disclosure. 

• Consult with graphic designers to improve the scorecard's layout. Better place- 
ment of text and the use of graphics can greatly improve the readability of the 
scorecard. The Obama administration should consult with graphic designers to 
create a visual hierarchy that leads the reader through the disclosure. 

• Test ways of better communicating the term "net price." Language like “out-of- 
pocket costs” or “net price” often does not convey the intended meaning. The 
government should test different ways of describing this concept. 

• Highlight the likelihood of graduating in four years, not six. Emphasis on the 
four-year graduation rate may have a bigger impact on students’ decisions, 
given that their college choices are based around an assumption that they will 
finish in that timeframe. 

• Develop alternative measures of student debt that matter to students. These 
focus groups seem to suggest that the traditional measures of student debt, such 
as repayment rate or default rate, are not meaningful to students. Average debt 
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loads are helpful but this number must be put into context with other measures 
that suggest to students the difficulty they may face in repaying their college 
debt. The average monthly payment figure should be prominently displayed and 
the government should consider including additional measures if they prove 
meaningful, including the average number of years graduates are in repayment. 

• Use links to provide additional employment outcomes. Students want to know 
how graduates fare in the job market but they want details. The government 
should explore ways to collect more information on employment outcomes. The 
college scorecard can include links to expanded outcomes, like graduates’ aver- 
age salary and employment outcomes by major or department. 

• Customize whenever possible. Averages can be helpful but only to a point. To 
make better decisions, students want college information that reflects their personal 
circumstances, including family income and major. The disclosure should provide 
links to customized information — for example, the net price calculator. The govern- 
ment should also devise ways to promote other forms of scorecard customization. 

It’s worth noting that the recommendations above may well necessitate changes 
in other education efforts by the government targeting prospective college stu- 
dents and their families. The U.S. Department of Education, for example, should 
consider emphasizing the four-year graduation rate rather than the six-year rate 
in other arenas where it reports to the public the success rates of publicly funded 
institutions — even if that makes campuses appear less successful. Our focus 
groups strongly suggest that students want consistency and comparability across 
communication platforms and initiatives. 
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Conclusion 


Designing a one-page “scorecard” featuring just a few pieces of data seems like a 
simple task. Certainly, there is no denying that the people who advise the White 
House on higher education policy are all highly capable people who study college 
students, went to college, and may have or know college-bound children. It would 
be easy for any of us, coming at this issue with similar backgrounds, to assume that 
we intuitively know what’s going on in the heads of the “customer” — after all, we 
are the customers. Moreover, why spend money on focus groups and surveys to 
tell us what we already know? 

The results described in this paper, however, show precisely why the federal gov- 
ernment must invest in rigorous design and testing processes for major consumer 
disclosures. Even the most well-meaning and conscientious policymakers cannot 
design a perfect disclosure without help from experts in communications and 
design, as well as, in this case, feedback from students. 

Our college scorecard focus groups need to be duplicated and expanded to 
generate a more definitive evaluation of the disclosure. Also, these focus groups 
should be accompanied by one-on-one cognitive interviews with both students 
and parents. But even our admittedly flawed focus group effort raised some real 
questions about how effectively the college scorecard will help students make 
better choices about college. 

In addition to the specific recommendations listed above, here are some general 
recommendations that federal policymakers should consider when mandating 
disclosures designed to improve the decision-making abilities of the public: 

• When Congress requires a disclosure by law, it should prescribe the goal of the dis- 
closure but not its contents. Rather, Congress should give the relevant agency the 
authority to develop and to complete consumer testing, requiring a report back 
that evaluates whether the finished product adequately fulfills Congress’s goals. 
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• The White House should issue an executive order asking all agencies to periodically 
conduct user testing of major consumer disclosures, and redesign them accordingly 

• The federal government should publicly release online any research on disclo- 
sure design and efficacy as it is developed, thereby building an online library of 
the latest guidelines and research on disclosure design. Likewise, all congressio- 
nally required reports on disclosure efficacy should be publicly released online. 
In addition to public release, this information should be published on a publicly 
accessible government website. 

• Whenever possible, agencies should use standard terms in disclosures to pro- 
mote better understanding of confusing but commonly used terms. 

To be sure, there is an upfront cost to conducting consumer research. But it’s a 
small investment that pales in comparison to the costs to society and taxpayers 
alike when vulnerable students make expensive wrong decisions about which col- 
lege to attend, and end up in default or worse. 

Ultimately, a well-designed “college scorecard” has the potential to nudge a stu- 
dent and his or her family to think hard about things that campus marketers don’t 
necessarily emphasize when they’re selling “collegiate fantasies” — issues students 
may neglect to carefully consider until it’s too late. And as the cost of college con- 
tinues to rise, these decisions become high-stakes for American students and their 
families. As one student remarked, “Gosh, all that money.” 
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Appendix: Principles of disclosure design 


When there is information asymmetry between buyers and sellers in a market, 
problems such as high prices and low quality often arise. Increasingly, the federal 
government is turning to disclosure as a means of addressing these problems. 
Disclosure can be an effective and less heavy-handed way of regulating an indus- 
try, but only if it is effective. 

Federal agencies need look no further than their own backyard for guidance on 
creating easy-to-understand disclosures. As part of the White House’s Open 
Government Initiative, the Office of Management and Budget’s Office for 
Information and Regulatory Affairs issued a memo in 2010 to the heads of all 
executive departments and agencies that instructs them on proper design of 
federal disclosures. 9 The memo provides a series of principles to be applied when 
preparing summary disclosures that, like the college scorecard, aim to help people 
make informed purchase decisions. 10 These principles include the following: 

• Identify the goals of the disclosure. 

• Avoid undue detail or complexity — be simple and specific. 

• Use accurate and plain language. 

• Properly place and time disclosures. 

• Use meaningful ratings or scales. 

• Test the disclosure in advance and monitor effects over time. 

• Consider the likely costs and benefits of disclosure. 

Many of these principles seem almost too obvious to mention. But there’s a reason 
former White House regulatory chief Cass Sunstein felt the need to include them 
in a government-wide memorandum — agencies often skip these basic steps and 
the result is needlessly complicated or irrelevant disclosures. 

For more detailed advice on disclosure design, we looked to the lessons learned by 
federal agencies that have designed and tested disclosures for years. These lessons 
are divided into design and content considerations. 
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Design 


Research shows that the design and layout of a disclosure can be just as impor- 
tant as its content. An effective and appealing layout leads the reader through the 
disclosure, drawing his or her eye to the most important points and presenting the 
content in a way that makes it easily understood. 

Years of consumer testing of disclosures by the Federal Reserve Board under- 
scores the importance of good design. The Fed found, for example, that the visual 
presentation of information such as credit card fees and interest charges can influ- 
ence consumer comprehension of those concepts. Fed researchers also found that 
design can affect whether consumers even look at a disclosure — layouts that look 
confusing are less likely to be read . 11 

The Kleimann Communication Group, a design agency frequently hired by federal 
agencies to conduct consumer disclosure research, offers on its website these 
general tips for good design: 

• Create visual unity. Disclosures need a cohesive design in which all the parts are 
visually consistent. 

• Use white space effectively. When the layout is too dense, text can be difficult 
to read or uninviting. 

• Designate a single focal point. Every printed product should have one obvious 
starting point and a clear navigational path. 

• Build in a hierarchy. Headings, subheadings, bullets or numbers reveal what is 
most important and least important. 

• Create balance. Alayout should be symmetrical to ensure that no single element 
dominates the disclosure. 

• Choose the right colors, grids, and typography. These formal elements of design 
can help create a sense of cohesiveness and highlight the most important points . 12 

We tend to think of disclosures as a printed publication, but the increasing use of 
electronic media should also affect design choices. The college scorecard may ben- 
efit from different design choices if presented online or in print. An online version, 
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for example ; might rely on links to additional information; but this may not make 
sense in a paper disclosure. 

The prospect of an online scorecard raises another design question: whether 
the scorecard should be customized. The sample scorecard posted on the White 
House’s website could be used as-is, with only one version of the form for each 
college or university. But it could also be customized in many ways, particularly 
in an online version. A student; for example; could enter basic information about 
family income, state of residence, and likely attendance pattern (part time or full 
time) to view a scorecard that is tailored to the average costs and benefits for 
similarly situated individuals. Or the scales used to show where a university falls 
compared to other universities could be customized according to a student’s pri- 
orities. These might display how University X compares to other colleges within a 
geographic region, or how it compares to institutions of similar size or selectivity. 

The decision to customize the form has both benefits and risks. Customization 
may mean that students are more likely to use the scorecard because it is more 
relevant to their decision making. But customization also greatly complicates the 
development process, risking the development of the scorecard becoming bogged 
down in technical details or that the disclosures are relegated to a federal website 
where families will never find them. 

The Federal Reserve Board’s consumer testing showed that some standardiza- 
tion is important to helping consumers recognize disclosures and compare across 
products. Also, the use of standard terms and formats across disclosures can help 
promote “transfer of learning” from one disclosure to another . 13 But when testing 
credit card disclosures, the Fed found that some consumers prefer to see standard 
disclosures placed in the context of their specific financial realities by showing 
how long it would take them to pay off their card balance or how much one would 
have to pay to wipe out the balance in three years. 


Content 

As important as design can be, a good layout is nothing without easily understood 
content. Plain spoken but meaningful content is key to a disclosure’s success — 
which is why crafting the written text is often the most difficult part of develop- 
ment. As with design, there are some helpful guidelines that policymakers should 
use to formulate content that’s most likely to assist consumers. The most impor- 
tant points are having a well-defined goal and writing in plain English. 
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Clearly identify the purpose 


The first big question a designer must ask before writing a single word is “What 
is the purpose of this disclosure?” Without clearly articulated goals, a disclosure 
designer may omit key information. Or he or she may get carried away trying to 
include data that are relevant to the topic but not necessary to achieve the govern- 
ment’s purpose. 

When the Department of Housing and Urban Development redesigned its Good 
Faith Estimate form, it began with clearly defined goals: to facilitate comparison 
shopping for mortgages; to make the costs of a mortgage clear; and to highlight 
often overlooked concepts such as yield spread premium, prepayment penalties, 
and balloon payments. 14 These goals drove the design and the data elements con- 
tained in the form, (see Figure 10) 


FIGURE 10 

The Good Faith Estimate form 

Design and data elements contained in the form were driven by clearly defined goals 
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The goals of the college scorecard are a bit harder to pinpoint. It has been 
described by the White House as a tool to help students choose the “right college 
for them.” 15 But the data used in the scorecard — average college cost, loan repay- 
ment, and earnings potential — also suggest a different, if related, purpose. These 
pieces of information are more suited to giving families “buyer beware” informa- 
tion that will help them avoid poor decisions, rather than sufficient information to 
help them find the “right college.” 

Though no single statement of purpose exists for the scorecard, the White House’s 
various statements on the scorecard project suggest there are multiple goals, 
including: 

• Improving students’ decision making by providing clear information about the 
costs and likely benefits of a college degree 

• Holding schools accountable for their tuition and outcomes 16 

• Educating students and parents about the prevalence of low graduation rates 
and high student debt levels 

• Suggesting the types of information that should be relevant to the college 
decision-making process but are often overlooked 


Use plain language 

Once the purpose of a disclosure is established, an agency must work to explain 
the concepts it seeks to convey in an easy-to-understand way. As the White House 
regulatory affairs office noted in a memo on government disclosures, people have 
limited time and attention. 17 Information must be presented in a way that best 
facilitates careful reading and processing. 

Disclosures can achieve this in two ways. First, the concepts presented should be as 
simple as possible. In trying to present concepts that are overly complex, a govern- 
ment agency increases the risk that consumers will misunderstand the disclosure 
or simply overlook it. Secondly, the language used to convey a concept should be 
plain: “Summary disclosure should avoid jargon, technical language, or extraneous 
information,” former White House regulatory chief Sunstein wrote in a 20 1 0 memo. 
“Each of these is distracting and threatens to turn away or to confuse users.” 18 
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Plain writing isn’t just a good idea — it’s also the law. Under the 2010 Plain Writing 
Act ; government communication to the public must be “clear; concise, well-orga- 
nized,” and presented in a way “that the public can understand and use.” 19 

To be sure, the concepts that a government agency seeks to disclose maybe 
unavoidably complex. In that case, the Kleimann Group suggests two techniques 
to help with comprehension. 20 First, Kleimann suggests that disclosures use 
“advance organizers,” or introductions and overviews that preview the elements 
readers will encounter. The Good Faith Estimate form, for example, begins with 
an “About Your GFE” section that answers two questions a reader may ask upon 
seeing the form: “What is a GFE?” and “How should you use this GFE to shop for 
the best loan?” 

Kleimann also suggests that if jargon cannot be avoided, key terms should be 
explained early in the document. Often, a key goal of financial disclosures is ensur- 
ing that consumers understand the jargon that is so often associated with products 
like credit cards, mortgages, or personal loans. The Federal Reserve’s consumer 
testing helped the agency identify instances where a term needed to be defined, 
changed, or even eliminated in order to make the disclosure readable and coher- 
ent. In a 201 1 bulletin, Fed researchers explained one such instance: 

Board researchers tried several ways to explain credit card “grace period" - how 
long a consumer has to pay the bill in full each month to avoid paying interest 
on purchases once the billing cycle closes. Some participants thought the term 
referred to the time they had after the payment due date before a late fee would 
be charged. After testing, the model disclosures were modified to use the language 
“how to avoid paying interest on purchases" instead of “grace period ." 21 

The draft college scorecard 
epitomizes both the need for 
plain language and the chal- 
lenges of simplification. The 
terminology used to explain 
“college price” is a bit unclear 
and is just one example. 

This section begins with a 
simple question: “What will 
it cost to attend UUS?” The 


FIGURE 1 1 

College costs on the government scorecard 

Students were substantially confused over cost terminology 
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question is quickly followed by a succession of terms such as “tuition and fees/’ 
“total costs/’ and “net price.” Each of these terms maybe unclear to a prospective 
student. What are the “fees”? Do “total costs” include room and board, travel, 
and book expenses? And while the scorecard includes a definition for net price — 
“average total costs after grants and scholarships” — this explanation may be too 
vague to properly relay the concept. 

There is more vague language 
in the area describing student 
loan repayment rates. The 
explanatory sentence in this 
section reads, “Former stu- 
dents are successfully repaying 
XX% of the total amount of 
federal student loans they took 
out to attend UUS.” 

It’s unclear what this means or how it would help a prospective student think 
about his or her own likelihood of repaying student loans on time. The description 
used with the student loan repayment scale is even more unclear. 

Even student loan experts participating in White House-led discussions struggled 
to understand the meaning of “Percentage of total loan amounts being repaid by 
former students compared to other institutions.” 

To simplify the scorecard language, designers must put themselves in the shoes of 
students and parents who have no experience with the college application process 
and explain concepts in terms that would resonate with those individuals. The 
designers should consider whether jargon should be explained or eliminated all 
together. If it is essential to the purpose of the scorecard that consumers learn a 
new term, then designers should strive to explain that term as clearly as possible. If 
a piece of jargon is only being used because it is a term of art in the financial aid or 
higher education policy world, it should be eliminated and replaced with a plain 
language description. 


FIGURE 12 

How the government scorecard addresses the likelihood of loan repayment 

Students did not understand the "student loan repayment" measure 
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Additional considerations 


There are no doubt many ways to improve the content of a government disclosure, 
but a few additional guidelines stand out as particularly important in summary 
disclosures such as the college scorecard. 


Design for usability 

When government agencies design disclosures, it is important to remember that 
disclosures should be both readable and usable. The form and content should be 
designed with comprehension in mind, but disclosures should also be evaluated 
based upon how well consumers can apply the content to their decision-making 
process. Plainer language will certainly make the college scorecard more readable, 
but the likelihood of influencing decisions may depend more on the content it 
conveys than on the words used. 


Relate the parts to the whole 

Like the college scorecard, many disclosures divide their information into three or 
four sections. For maximum comprehension and usability, designers must ensure 
that all parts of the disclosure relate to one another and advance the overall mes- 
sage of the form. One way to achieve this is to focus on the sequencing of informa- 
tion — creating a logical order and using contextual content to help readers make 
the connection. 


Consider changing the choice structure to fit the disclosure 

Often disclosures are a default solution helping consumers navigate a difficult 
decision process such as choosing a mortgage or a college. But sometimes the 
complicated nature of the choice can make simple disclosure nearly impossible. 
Designers should consider whether the best course of action is to make the choice 
simpler and not the disclosure. When the Fed tried to address bank overdraft 
procedures, for example, it found that the lack of standardization in bank practices 
made it difficult to create a simple disclosure. So they developed rules that would 
standardize bank practices, making the disclosure more straightforward . 22 
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The price of college should be a simple idea to convey in the scorecard. But it is 
made more complicated by tuition discounting, in-state rates, and the prolifera- 
tion of add-on fees. The Department of Education may find that it is better to 
work to simplify college-pricing practices rather than try to explain them. 
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